Skip to content

fix: skill metadata truncation must fall back to UTF-8 sequence boundary - #32

Merged
shaohaozhi286 merged 3 commits into
masterfrom
fix/skill-loader-utf8-truncation
Sep 2, 2026
Merged

fix: skill metadata truncation must fall back to UTF-8 sequence boundary#32
shaohaozhi286 merged 3 commits into
masterfrom
fix/skill-loader-utf8-truncation

Conversation

@LIUXIN557

Copy link
Copy Markdown
Collaborator

Behavior change

skill_loader's truncate() used raw-byte substr(0, n-3) + "...". When a skill's description / whenToUse exceeds the 1024-byte budget and the cut lands inside a multi-byte UTF-8 character (long Chinese descriptions, e.g. lark-apps at 1289 bytes), the result is an invalid sequence (orphan lead byte + 0x2E from "...").

That string enters the skills index message; nlohmann::json body.dump() then aborts the entire request with:

[Error] [json.exception.type_error.316] invalid UTF-8 byte at index 16944: 0x2E

Trigger conditions (all reproduced):

  • TUI only (headless does not inject the skills index message)
  • Large-context model presets (e.g. deepseek-v4-flash with 1M window -> full descriptions included; 128K presets truncate before the bad bytes)
  • Any skill with a >1024-byte description whose byte 1021 sits mid-character

Fix

  • Route truncate() through the existing truncate_utf8_prefix() helper so cuts fall back to a UTF-8 sequence boundary.
  • Guard final metadata with ensure_utf8() for defense in depth.

Tests

  • New tests/skills/skill_loader_utf8_test.cpp: description and whenToUse truncation with a multi-byte char at the budget boundary must stay valid UTF-8.
  • End-to-end: pty-driven TUI + --resume on a deepseek session reproduced type_error.316 deterministically before the fix; after the fix the request serializes and sends normally.

Verification

cmake --build build/macos-x64-release --target acecode acecode_unit_tests
ctest / gtest filter SkillLoaderTruncation.*  # 2/2 passed

slyxyllt and others added 3 commits September 2, 2026 01:14
truncate() used raw-byte substr + "..." which can cut a multi-byte
character in half, leaving an invalid sequence (trailing lead byte +
0x2E) inside the skills index message. nlohmann body.dump() then aborts
the whole request with type_error.316 (invalid UTF-8). Route truncation
through truncate_utf8_prefix and guard the result with ensure_utf8.

Reproduced with deepseek-v4-flash (1M context) TUI session: full skill
descriptions enter the request, any skill with a >1024-byte Chinese
description (e.g. lark-apps) triggered the error deterministically.

Add regression tests for description/whenToUse truncation.
@shaohaozhi286
shaohaozhi286 merged commit 5a478ec into master Sep 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants